Unimodal Loading Problems

This paper deals with optimal learning and provides a uniied viewpoint of most signiicant results in the eld. The focus is on the problem of local minima in the cost function that is likely to aaect more or less any learning algorithm. We give some intriguing links between optimal learning and the computational complexity of loading problems. We exhibit a computational model such that the solution of all loading problems giving rise to unimodal error functions require the same time, thus suggesting that they belong to the same computational class. 1 Learning as optimisation Supervised learning in multilayered networks (MLNs) can be accomplished thanks to Backpropagation (BP), which is used to minimise pattern misclassiications by means of gradient descent for a particular nonlinear least squares tting problem. Unfortunately, BP is likely to be trapped in local minima and indeed many examples of local extremes have been reported in the literature. The presence of local minima derives essentially from two diierent reasons. First, they may arise because of an unsuitable joint choice of the functions which deenes the network dynamics and the error function. Second, local minima may be inherently related to the structure of the problem at hand. In 5], these two cases have been referred to as spurious and structural local minima, respectively. Problems of sub-optimal solutions may also arise when learning with high initial weights, as a sort of premature neuron saturation arises, which is strictly related to the neuron fan-in. An interesting way of facing this problem is to use the \relative cross-entropy metric " 10], for which the erroneous saturation of the output neurons does not lead to plateaux, but to very high values of the